Content-adaptive multi-focal display

ABSTRACT

A multi-focal display represents a 3-dimensional scene by a series of 2-dimensional images located at different focal planes. The locations of the focal planes are selected based on an analysis of the three-dimensional scene to be rendered by the multi-focal display.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority under 35 U.S.C. §119(e) to U.S.Provisional Patent Application Ser. No. 62/084,264, “Content-AdaptiveMulti-Focal Display,” filed Nov. 25, 2014. The subject matter of all ofthe foregoing is incorporated herein by reference in its entirety.

BACKGROUND

1. Field of the Invention

This disclosure relates generally to multi-focal displays.

2. Description of Related Art

Multi-focal displays (MFDs) typically use rapid temporal and focalmodulation of a series of 2-dimensional images to render 3-dimensional(3D) scenes that occupy a certain 3D volume. This series of images istypically focused at parallel planes positioned at different, discretedistances from the viewer. The number of focal planes directly affectsthe viewers' eye accommodation and 3D perception quality of a displayedscene. If a given 3D scene is continuous in depth, too few planes maymake the MFD rendering look piecewise with discontinuities betweenplanes or result in contrast loss. More planes is typically better interms of perceptual quality, but can be more expensive to implement andoften may not be achievable because of practical display limitationsincluding bandwidth and focal modulation speed.

Therefore, an important consideration for MFDs is the focal planeconfiguration, including the number of focal planes and the location ofthe focal planes (that is, distances from the viewer). Multi-focaldisplays typically use focal plane configurations where the number andlocation of focal planes are fixed. Often, the focal planes areuniformly spaced. This one size fits all approach does not take intoaccount differences in the scenes to be displayed and the result can bea loss of spatial resolution and perceptual accuracy.

Thus, there is a need for better approaches to determining focal planeconfiguration.

SUMMARY

The present disclosure overcomes the limitations of the prior art byselecting the locations of the focal planes for a multi-focal display,based on an analysis of the scene to be rendered by the multi-focaldisplay. In one example, a distortion metric is defined that measures adistortion between an ideal rendering of a three-dimensional sceneversus the rendering by a limited number of focal planes in themulti-focal display. The locations of the focal planes are selected byoptimizing the distortion metric. One distortion metric is based ondifferences between the location of a point in the ideal renderingversus the location of the closest focal planes of the multi-focaldisplay. Another distortion metric is based on differences in thedefocus blurring for the ideal rendering versus the rendering by themulti-focal display.

Other aspects include components, devices, systems, improvements,methods, processes, applications, computer readable mediums, and othertechnologies related to any of the above.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the disclosure have other advantages and features whichwill be more readily apparent from the following detailed descriptionand the appended claims, when taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram of a multi-focal display according to the presentinvention.

FIG. 2 is a histogram of z locations from a 3D scene, overlaid withfocal plane locations for uniform focal plane spacing, K-means focalplane spacing and weighted K-means focal plane spacing.

FIGS. 3a-3d are images showing the effect of different types of focalplane spacing.

FIG. 4 is a plot of a depth-blended defocus transfer function.

FIG. 5a plots the accommodation state that maximizes the metric βagainst input spatial frequency. FIG. 5b plots (β_(max)−β_(min))/β_(max)against spatial frequency.

FIGS. 6a-6c show simulated eye responses for stimulus with differentspatial frequencies rendered between planes using depth blending.

FIGS. 7a-d are diagrams showing different types of multi-focal displays.

The figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesdescribed herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Introduction

FIG. 1 is a diagram of a multi-focal display 100 according to thepresent invention. The MFD 100 includes a display 110, an adjustableoptical element 120 and modules 130-160 for scene rendering and focalplane control. Examples of optical element 120 include deformablelenses, lenses with adjustable index of refraction, and deformablemirrors. Modules 130-160 could be implemented in hardware, software or acombination of the two. The optical element 120 is adjustable. Atdifferent adjustments, the display 110 appears at different locations(focal planes), which are represented by the dashed lines in FIG. 1. Inthis way, a 3D scene can be approximated by a series of 2D imagesrendered at the different focal planes.

Optional pre-processing module 130 receives data representing the 3Dscene to be rendered and adapts it to rendering requirements. Forexample, pre-processing module 130 may perform functions such asmagnifying, cropping and sharpening. Focal plane placement module 140analyzes the content of the 3D scene and selects the locations of thefocal planes based on the content analysis. The selection can also bebased on rendering requirements. Scene separation module 150 separatesthe 3D scene into the constituent 2D images to be rendered. Thistypically involves depth blending, as will be described below. Thecontent of each 2D image will depend on the focal plane locations.Rendering engine 160 then renders the 2D images onto the display, incoordination with adjustment of the optical element 120 to effect thedifferent focal planes. Additional post-processing can also beperformed. For example, smoothing constraints (temporal and/or spatial)may be applied, or occlusion edges may be processed to further improveperceived quality.

In FIG. 1, the MFD dynamically adjusts the focal plane settings based onthe content of the scene and/or rendering requirements, for example tominimize contrast loss attributed to depth blending and/or to maximizethe perceptual quality of the rendered 3D scene. The focal planes neednot be uniformly spaced. Nor are they required to be statically located.The locations can be dynamically adjusted depending on the scene contentand/or rendering requirements. For example, the latest DMD (digitalmicromirror device) chips used in multi-focal displays can achieve aflicker-free display by multiplexing about 6 focal planes at 60 Hz perplane. In this case, a viewer can view the displayed 3D scene andcorrectly accommodate to scene content at those six planes. This numberof focal planes is typically sufficient for single-user, near-the-eyemulti-focal displays. This speed is sufficient to render video inreal-time. GPUs may be used to speed up calculation. The focal planeconfiguration may be adjusted for each frame of video or lessfrequently, for example every certain number of frames or for eachscene.

Depth Blending

MFD technology can represent a 3D scene by a series of 2D images atdifferent focal planes due to a concept known as depth blending. Byilluminating two adjacent focal planes simultaneously, a focus cue maybe rendered at any axial distance between the planes. Since the twofocal planes lie along a line of sight, the luminance provided by eachof the adjacent focal planes determines where the cue will be highest(where the eye perceives the highest visual quality, or where the areaunder the modulation transfer function (MTF) observed by the eye ishighest).

A simple form of luminance weighting used for depth blending is a linearinterpolation of the luminance values observed by each pixel for theadjacent focal planes, which we will use as an example although othertypes of depth blending can also be used. Let w_(n) and w_(f)respectfully denote the luminance weights given to the near and farfocal planes. These values, which sum to 1 to retain the correctluminance perceived by the eye, are computed as follows:

$\begin{matrix}{{w_{f} = \frac{z_{n} - z}{z_{n} - z_{f}}},} & (1) \\{w_{n} = {1 - {w_{f}.}}} & (2)\end{matrix}$where z_(n) and z_(f) are the locations of the near and far focal planesand z is the actual location of the object in the 3D scene, which isbetween z_(n) and z_(f). In this linear formulation, if z=z_(n) (objectpoint at the near focal plane), then w_(f)=0 and w_(n)=1, meaning thatall of the luminance is allocated to the near focal plane. Conversely,if z=z_(f) (object at the far focal plane), then w_(f)=1 and w_(n)=0,and all of the luminance is allocated to the far focal plane. For anintermediate position such as z=(z_(n)+z_(f))/2, then w_(f)=½ andw_(n)=½ so luminance is split between the far and near focal planes. Inthis way, a virtual object can be rendered at any position z betweenz_(n) and z_(f) by splitting its luminance between the two imagesrendered at focal planes z_(n) and z_(f).

Problem Formulation

We first formulate the problem of placement of focal planes based on agiven objective function, and then show two examples of differentobjective functions. The objective function typically is a type ofdistortion metric that measures a distortion between an ideal renderingof the 3D scene versus the rendering by the MFD.

Let (x,y,z) denote the two transverse dimensions and the axial dimensionof the 3D space rendered by the MFD. In practice, what we are typicallygiven are the following quantities:

-   -   an N-voxel 3D scene to be projected S={(p_(n), I_(n)), n=1, . .        . , N}, where p_(n)=(x_(n),y_(n),z_(n)) denotes a vector of 3D        coordinates of a 3D point, and I_(n) denotes the intensity or        color value of that 3D point. These points can be obtained by a        3D camera or generated by a computer graphics engine, for        example.    -   number of available depth planes M        Given these quantities, we want to estimate the following        unknown variables:    -   position of focal planes q=(q₁, q₂, . . . , q_(M)). Note that        the values q_(m) are actually z-coordinates of focal planes and        that the focal planes are fronto-parallel to the eye. We use q        instead of z to clearly separate the focal plane positions from        other z values.

To estimate the best positions of focal planes, we formulate thefollowing optimization problem:

$\begin{matrix}{{find}{q^{*} = \left( {q_{1},{q_{2\mspace{14mu}}\ldots}\mspace{14mu},q_{M}} \right)}} & (3) \\{{{such}\mspace{14mu}{that}}{{q^{*} = {\arg\;{\min\limits_{q}\;{D\left( {S,q} \right)}}}},}} & (4)\end{matrix}$where the objective function D(S, q) denotes a distortion error metricfor representing a 3D scene S on M focal planes positioned at q=(q₁, q₂,. . . , q_(M)). This can in general be any metric that minimizes theerror compared to a perfect rendering.

Alternately, we can pose the optimization problem such that it finds asolution for focal plane placement that maximizes the quality of the 3Dscene rendering Q(S, q):

$\begin{matrix}{{find}{q^{*} = \left( {q_{1},{q_{2\mspace{14mu}}\ldots}\mspace{14mu},q_{M}} \right)}} & (5) \\{{{such}\mspace{14mu}{that}}{{q^{*} = {\arg\;{\max\limits_{q}\;{Q\left( {S,q} \right)}}}},}} & (6)\end{matrix}$

In the following, we show two specific examples of automatic focal planeplacement. In the first example, we use an error metric D(S,q) andminimize it to obtain q. In the second example, we use a quality metricQ(S,q) that can be used for focal plane placement. Other distortionmetric functions, including other error or quality metrics, can be usedas well.

Solution Example 1: Focal Plane Placement Based on 3D Point Clustering

The first example of an objective function can be derived by consideringthe problem of focal plane placement as a clustering problem. Given thez-coordinates of all 3D data points in a scene. That is, given z₁, z₂, .. . , z_(N), we can use the K-means algorithm to find the best placementof M focal planes. In this case, our optimization problem becomes:

$\begin{matrix}\begin{matrix}{q^{*} = {\arg\;{\min\limits_{q = {({q_{1},{q_{2}\ldots},q_{M}})}}{D_{KM}\left( {S,q} \right)}}}} \\{= {\arg\;{\min\limits_{q = {({q_{1},{q_{2}\ldots},q_{M}})}}{\frac{1}{MN}{\sum\limits_{n = 1}^{N}{\sum\limits_{m = 1}^{M}{{z_{n} - q_{m}}}_{2}^{2}}}}}}}\end{matrix} & (7)\end{matrix}$

Solving this problem using the K-means algorithm gives a placement offocal planes such that the focal planes used to represent 3D data areclose to the actual location of the data. Hence, in most cases thisoptimization problem will give a solution different from theconventional strategy of uniform focal plane spacing. Note that in theoptimization above, instead of distance z in meters, we can also usedistance in diopters (inverse meters) or other measures of opticalpower, in order to take into account for the decreasing sensitivity ofdepth perception with increasing distance.

Spatial frequencies of the content also impact accommodative responsewhen depth blending is used. For low-frequency stimuli (for example, 4cycle per degree or cpd), linear depth blending can drive accommodationrelatively accurately between planes. But for high-frequency stimuli(for example, 21 cpd) and broadband stimuli (for example, 0-30 cpd),accommodation is almost always at or near a focal plane no matter howthe luminance weights w_(f), w_(n) are distributed. Therefore, aweighted K-means algorithm can be used to take this spatial frequencydependency into account. For example, if the spatial frequency orspatial gradient value near a point is higher than a threshold, it canbe assigned a large weight, otherwise it can be assigned a small weight.Denote ω _(n) as the weight associated with each data point, Eq. 7 canbe adapted to:

$\begin{matrix}\begin{matrix}{q^{*} = {\arg\;{\min\limits_{q = {({q_{1},{q_{2}\ldots},q_{M}})}}{D_{KM}\left( {S,q} \right)}}}} \\{= {\arg\;{\min\limits_{q = {({q_{1},{q_{2}\ldots},q_{M}})}}{\frac{1}{MN}{\sum\limits_{n = 1}^{N}{\sum\limits_{m = 1}^{M}{{\overset{\_}{\omega}}_{n}{{z_{n} - q_{m}}}_{2}^{2}}}}}}}}\end{matrix} & (8)\end{matrix}$

FIG. 2 shows experimental results using the K-means and weighted K-meansfocal plane allocation algorithms described above. FIG. 2 shows ahistogram of actual z locations from the 3D chess scene shown in FIG. 3a. FIG. 3b shows the same z locations as a grayscale image. In thisparticular example, the 3D scene has some but fewer points in the range(+1.0,+1.6)D, and then denser distribution of points in the range(+1.6,+2.0)D. The density in the latter range is because the scenecontains a limited number of discrete chess pieces, each of which islocated at a different depth.

Table 1 below shows the focal plane positions using uniform focal planespacing, using K-means focal plane spacing and using weighted K-meansfocal plane spacing.

TABLE 1 Focal plane locations (in diopters) Uniform K-means WeightedK-means +0.00 +1.00 +1.00 +0.60 +1.20 +1.30 +1.20 +1.46 +1.57 +1.80+1.64 +1.81 +2.40 +1.82 +1.90 +3.00 +2.00 +2.00These focal plane locations are also shown by the arrows above the graphin FIG. 2. The uniform configuration was chosen according to theliterature. It is evenly spaced from 0 D to +3.00 D to accommodate avariety of different scenes. However, this scene only spans +1.00 D to+2.00 D, so many of the focal planes are wasted. As can be seen, thecontent-adaptive algorithms allow focal planes to adapt to content depthdistribution and concentrate focal planes where there is data. Incomparison, uniform focal plane spacing is content-agnostic, which canresult in more contrast loss.

FIGS. 3a-3d are images showing the effect of different types of focalplane spacing. We use these images to compare uniform focal planespacing and adaptive focal plane spacing. FIG. 3 a shows the input 3Dscene and FIG. 3b shows the depth map of the 3D scene in diopters. Thebishop (indicated by the arrow in FIG. 3a ) is the simulatedaccommodation target at approximately 1.63 D. FIG. 3c shows a simulatedretinal image when the 3D scene is rendered by a six-plane MFD, wherethe focal planes are uniformly spaced as shown in Table 1 above. FIG. 3dshows a rendering, where the focal plane locations are determined usingK-means clustering. Note that the rendered image in FIG. 3d appears moresharply focused than that of FIG. 3c because the bishop is closer tofocal planes placed with the K-means algorithm than it is to thoseplaced with uniform spacing.

K-means is used just as an example. Other clustering techniques can beapplied, for example clustering based on Gaussian Mixture Models (GMM)or support vector machines (SVM).

Solution Example 2: Focal Plane Placement Based on Defocus Metric

When a given 3D scene with continuous depth values is displayed on amulti-focal display with a finite number of focal planes, human eyeswill perceive it with a certain amount of defocus compared to an idealcontinuous 3D rendering. We describe here a model of that defocus, whichwe then use within our objective function for focal plane placement.Namely, our objective function will place the focal planes such that itmaximizes the quality of the 3D scene rendering by minimizing thedefocus.

Optical defocus is typically modeled through Fourier optics theory, in acontinuous waveform domain. Therefore, assume that a given 3D scene is aset of samples from a continuous 3D function ƒ(x,y,z), where we havethat I_(n)=ƒ(x_(n),y_(n),z_(n)) for n=1, 2, . . . , N given points inour 3D scene. We first provide a Fourier derivation of a human eye'ssensitivity to defocus and then use the derived theory to define aquality metric for a given 3D scene.

Let primed coordinates (x′, y′) denote the retinal coordinates. When theeye accommodates to a distance z_(e), a 2D retinal image g(x′, y′) maybe expressed as a convolution of the 3D object with the 3D blur kernelh(x, y, z) evaluated at a distance z_(e)−z, followed by integrationalong the axial dimension:g(x′,y′,z _(e))=∫∫∫f(x,y,z)h(x−x′,y−y′,z _(e) −z)dxdydz.  (9)Note that in the case of in-focus plane-to-plane imaging (z_(e)−z=0),the convolution kernel h reduces to the eye's impulse response. Thisconfiguration yields maximum contrast, where contrast is defined in theconventional way in the spatial frequency domain. Deviations from thatin-focus imaging result in a reduction in contrast. The severity of thelost contrast depends on the amount of defocus.

To quantify the effects of defocus, we turn to the pupil function of theeye's optical system. For a rotationally-symmetric optical system withfocal length F and circular pupil of diameter A, the lens transmittancethrough the exit pupil is modeled as:

$\begin{matrix}{{{t\left( {x,y} \right)} = {\exp\frac{- {{\mathbb{i}\pi}\left( {x^{2} + y^{2}} \right)}}{\lambda\; F}{P\left( {x,y} \right)}}},} & (10)\end{matrix}$where the pupil function P is given by

$\begin{matrix}{{P\left( {x,y} \right)} = {{{circ}\left( {\frac{x}{A},\frac{y}{A}} \right)}.}} & \;\end{matrix}$In our system, the pupil diameter A may vary between ˜2-8 mm based onlighting conditions. Though the eye is, in general, not rotationallysymmetric, we approximate it as such to simplify formulation in thisexample.

In the presence of aberrations, the wavefront passing through the pupilis conventionally represented by the generalized pupil function G(x,y)=P(x, y)exp(iφ(x, y)), where the aberration function φ is a polynomialaccording to Seidel or Zernike aberration theory. The defocus aberrationis commonly measured by the coefficient w₂₀ of φ. Defocus distortion canalternatively be modeled by including a distortion term θ_(z) in thepupil function and defining the pupil function of a system defocused bydistance θ_(z) in axial dimension asP _(θ) _(z) (x,y)=exp(πi(θ_(z)/λ)(x ² +y ²))P(x,y),  (11)where θ_(z)=1/z+1/z_(r)−1/F with z_(r) being the distance between thepupil and the retina. The relationship between θ_(z) and theconventional defocus aberration coefficient w₂₀ is given byθ_(z)=2w₂₀/A². Using this formulation, we can formulate the defocustransfer function, which is the optical transfer function of thedefocused system, as the auto-correlation of the pupil function of thedefocused system as follows:

$\begin{matrix}{{{\hat{h}}_{\theta_{z}}\left( {u,v} \right)} = {\int{\int{{P_{\theta_{z}}^{*}\left( {{x - \frac{\lambda\; d_{r}u}{z}},{y - \frac{\lambda\; d_{r}v}{2}}} \right)}{P_{\theta_{z}}\left( {{x + \frac{\lambda\; d_{r}u}{2}},{y + \frac{\lambda\; d_{r}v}{2}}} \right)}{\mathbb{d}x}{\mathbb{d}y}}}}} & (12)\end{matrix}$Now we replace the defocus distortion distance θ_(z) with 1/z_(e)−1/zand define the normalized defocus transfer function (DTF) of the eye as

$\begin{matrix}{{\hat{H}\left( {u,v,z,z_{e}} \right)} = {\frac{{\hat{h}}_{{1/z_{e}} - {1/z}}\left( {u,v} \right)}{{\hat{h}}_{0}\left( {0,0} \right)}.}} & (13)\end{matrix}$Optical aberrations of the eye and/or the MFD system can be modeled intothe DTF as well.

The image as formed on the retina is described by the multiplication ofthe defocus transfer function and the Fourier transform of the functionƒ(u,v,z) describing the object displayed at distance z from the eye byĝ(u,v,z,z _(e))=Ĥ(u,v,z,z _(e)){circumflex over (ƒ)}(u,v,z).  (14)

In a MFD system, we can typically display only a small number of focalplanes fast enough to be perceived as simultaneously displayed by thehuman eye. For the case that two objects are being displayed at twofocal planes located at distances q₁ and q₂ away from the eye, the eyeintegrates the two objects as imaged through the eye's optical system.That is, it integrates over the light emitting from the two objectsafter passing through the eye's optical system described by the defocustransfer function. We derive this image formation at the retina plane bythe following formulaĝ _(r)(u,v,q ₁ ,q ₂ ,z _(e))=Ĥ(u,v,q ₁ ,z _(e)){circumflex over(ƒ)}(u,v,z)+Ĥ(u,v,q ₂ ,z _(e)){circumflex over (ƒ)}(u,v,z).  (15)If linear depth blending is applied to the input scene ƒ(x,y,z), usingcoefficients w₁ and w₂, then the Fourier transform of perceived image onthe retina is described byĝ _(r)(u,v,q ₁ ,q ₂ ,z _(e))=w ₁ Ĥ(u,v,q ₁ ,z _(e)){circumflex over(ƒ)}(u,v,z)+w ₂ Ĥ(u,v,q ₂ ,z _(e)){circumflex over (ƒ)}(u,v,z).  (16)Using this observation, we define the depth-blended defocus transferfunction of the entire system asĤ _(blend)(u,v,(q ₁ ,q ₂),z _(e))=w ₁ Ĥ(u,v,q ₁ ,z _(e))+w ₂ Ĥ(u,v,q ₂,z _(e)),  (17)

FIG. 4 shows this function for various levels of defocus {−0.3, −0.2, .. . +0.3}D. FIG. 4 plots the depth-blended defocus transfer function ofa 3 mm pupil observing a stimulus located at 1.5 D as rendered by twofocal planes located at 1.2 and 1.8 D. Curve 400 is the ideal MTF. Curve410 is the DTF for a defocus of OD, curve 411 is the DTF for a defocusof +0.1 D or −0.1 D, curve 412 is for defocus of +/−0.2 D, and curve 413is for defocus of +/−0.3 D. Note there is a spatial frequency (in thiscase approximately 18 cpd) at which the different DTF curves intersect.Spatial frequencies lower than this transitional frequency generate thecorrect focus cues. Above this frequency, the depth-blended defocustransfer function curve for 0 D of defocus is lower than that of +/−0.3D of defocus. For stimuli within this frequency range, the eye is forcedto accommodate at one of the adjacent focal planes rather than thetarget stimulus location, resulting in an incorrect focus cue.

We can also generalize this blending function using all display planesq₁, . . . , q_(M) to derive an effective or blended transfer functionfor the multi-focal display as:

$\begin{matrix}{{{{\hat{H}}_{blend}\left( {u,v,q,z_{e}} \right)} = {\sum\limits_{m = 1}^{M}{w_{m}{{\hat{H}\left( {u,v,q_{m},z_{e}} \right)}.{for}}}}}{q = {\left( {q_{1},\ldots\mspace{14mu},q_{M}} \right).}}} & (18)\end{matrix}$

Depth blending drives the accommodation of the eye to a focal plane witha Ĥ_(blend)(u, v,q, z_(e)) closest to the ideal DTF curve. We can seefrom FIG. 4 that this accommodation plane distance depends greatly onspatial frequency. Therefore, we use the theory developed above toderive a content-aware metric to quantify the impacts that focal planeplacement and depth fusion have on effective resolution loss.

The eye will accommodate to a distance that maximizes the area under theDTF. However, since that distance depends on the spatial frequency, wefurther assume that the eye will accommodate to the distance thatmaximizes a certain quality metric Q_(DM)(S,q) based on this defocusmeasure (area under the DTF). Since this distance varies with eachpatch, we seek a solution that incorporates all of the patches into asingle metric.

In one approach, we partition the displayed image ƒ(x,y,z) into N_(p)patches ƒ_(i)(x,y,z_(i)), i=1, . . . , N_(p), where z_(i) is a scalarrepresenting the i^(th) patch's mean object distance. Overlappingpatches may be used. We may compute each patch's Fourier transform andmultiply it with the depth-fused DTF to find the information transferredfrom a stimulus to the eye according to a placement of focal planeslocated at q={q₁, q₂, . . . , q_(M)} and a local stimulus located atdistance z_(o) to compute the scalar value β_(i) for each patch:β_(i)(z _(i) ,q)=∫_(u) ₀ ^(u) ¹ ∫_(v) ₀ ^(v) ¹ {circumflex over (ƒ)}_(i)(u,v,z _(i))Ĥ _(blend)(u,v,q,z _(o))dudv.  (19)where [u₀, u₁] and [v₀, v₁] denote the frequency interval of interest.Other metrics describing the object's information content, such asmeasures of contrast, entropy, or other transformative metrics could beused to define β_(i)(z_(i), q) as well.

If we store the metrics from all of the patches into a vector β we canalter the focal plane placement for up to M focal planes. We seek tosolve the following optimization problem to find q*, the optimal set ofdioptric distances to place the available focal planes:

$\begin{matrix}{q^{*} = {{\arg\;{\max\limits_{q}{Q_{DM}\left( {S,q} \right)}}} = {\arg\;{\max\limits_{q}{\sum\limits_{i = 1}^{N_{p}}{\beta\left( {z_{i},q} \right)}^{2}}}}}} & (20)\end{matrix}$which can be relaxed or adjusted if not solvable in realistic time.

The resulting entries of q* signify where best to place the set of Mfocal planes. For example, optimizing 2 focal planes to represent 3objects clustered about dioptric distances of 1/z₁=0.6 D, 1/z₂=1.5 D;1/z₃=2.0 D might result in the optimal focal plane placement of 1/q₁=1.1D, 1/q₂=1.8 D.

The solution for q could begin with an initial guess of uniform focalplane spacing based on the available focal planes. For example, a6-plane system seeking a workspace between 0 and 3 diopters could startwith {0, 0.6, 1.2, 1.8, 2.4, 3.0}D. As the optimization algorithmiterates through iterations k, the entries of q would change until|Q_(DM) ^(k)(S,q)−Q_(DM) ^(k+1)(S,q)|≦ε, where ε is a toleranceparameter telling the algorithm when to stop. Extra specifications couldbe incorporated into the optimization algorithm to constrain thefeasible solution set, as well.

Finally, note that the metric Q_(DM)(S, q) quantifies the quality of therendering of a given 3D scene, with respect to defocus. Therefore, inaddition to focal plane placement, this metric can be also used forrendering quality assessment in MFDs.

FIGS. 5-6 show simulation results for the approach described above. Thisexperiment validates the behavior of the metric β of Eq. 19. During theexperiment, two focal planes were set at distances 1/q₁=1.2 D, 1/q₂=1.8D. The stimulus, a set of cosine waves incrementing in spatial frequencyby 1 cpd, was simulated at a virtual distance 1/z_(o)=1.5 D away fromthe observer, or right between the two focal planes.

The eye's accommodation was varied in increments of 0.1 D between thesetwo focal planes. The accommodation is between −0.3 and +0.3 D, where +0D corresponds to the dioptric midpoint of the focal planes at q₁ and q₂.FIG. 5a plots the accommodation state that maximizes the metric βagainst input spatial frequency. FIG. 5b plots (β_(max)−β_(min))/β_(max)against spatial frequency, which should minimize at u=0 and u=18 cyclesper degree as shown in the depth-blended defocus transfer function plotsof FIG. 4. Other metrics can be used. These plots show that the metricwill be highest at the dioptric midpoint of the two focal planes forlower and middle spatial frequencies. When the local stimulus spectrumis above the transition frequency, the metric will maximize at one ofthe focal planes.

FIGS. 6a-6c show the simulated eye responses for stimulus with differentspatial frequencies rendered between planes using depth blending. FIG. 6a shows 7 squares which are images of a 9 cpd image. For each square inthe figure, the eye accommodates to the state shown in Table 2.

TABLE 2 Eye accommodations −0.3 D −0.2 D −0.1 D   0 D +0.1 D +0.2 D +0.3D Not used Not usedThat is, the top left square is an image of a 9 cpd image where the eyeaccommodates to −0.3 D. For the top middle square, the eye accommodatesto −0.2 D, and so on. The bottom middle and bottom right squares are notused, so they are left blank. FIGS. 6b and 6c show the same arrangementof eye accommodations, but for a 18 cpd and 25 cpd image, respectively.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. For example, FIG. 1 shows amulti-focal display with a finite number of planar focal planes that areall located to one side of the display, as reproduced in FIG. 7a . InFIG. 7a , the dashed box 700 represents the 3D focal volume to berendered and, in this example, it is rendered by images located at thefocal planes represented by the solid lines 710. In alternateembodiments, the focal planes could be distributed to both sides of thedisplay and they could be non-planar. For example, as shown in FIG. 7b ,there could be a number of focal surfaces 712, which are curved or haveother non-planar shapes. In addition, in FIG. 7c , the focal surfaces714 have different shapes. FIG. 7d shows an example where themulti-focal display can render points at more than a finite number ofsurfaces. In this example, 716 is a slice that has volume and themulti-focal display can render points within that volume. This is truefor each of the volumes shown. However, the volumes in the aggregate donot allow address of every point within the focal volume 700. That is,points that are located outside the slices will be represented by depthblending between different slices. For convenience, the term “renderablevolume” will be used to refer to both 2D surfaces as shown in FIGS. 7a-cand 3D volumes as shown in FIG. 7 d.

In another aspect, in addition to selecting the locations of therenderable volumes, the multi-focal display also selects the number ofrenderable volumes. In the original example with six focal planes, themulti-focal display might determine the number M of focal planes where Mcan be up to six. Less than the maximum number may be selected forvarious reasons, for example to reduce power consumption.

In yet another aspect, FIG. 1 shows a multi-focal display for one eye.Two-eye and stereo systems can also be used. In addition, additionaloptics, such as beamsplitters, may be used to combine the scene renderedby the multi-focal display with other scenes or the surroundingenvironment.

Various other modifications, changes and variations which will beapparent to those skilled in the art may be made in the arrangement,operation and details of the method and apparatus of the presentinvention disclosed herein without departing from the spirit and scopeof the invention as defined in the appended claims. Therefore, the scopeof the invention should be determined by the appended claims and theirlegal equivalents.

In alternate embodiments, aspects of the invention are implemented incomputer hardware, firmware, software, and/or combinations thereof.Apparatus of the invention can be implemented in a computer programproduct tangibly embodied in a non-transitory machine-readable storagedevice for execution by a programmable processor; and method steps ofthe invention can be performed by a programmable processor executing aprogram of instructions to perform functions of the invention byoperating on input data and generating output. The invention can beimplemented advantageously in one or more computer programs that areexecutable on a programmable system including at least one programmableprocessor coupled to receive data and instructions from, and to transmitdata and instructions to, a data storage system, at least one inputdevice, and at least one output device. Each computer program can beimplemented in a high-level procedural or object-oriented programminglanguage, or in assembly or machine language if desired; and in anycase, the language can be a compiled or interpreted language. Suitableprocessors include, by way of example, both general and special purposemicroprocessors. Generally, a processor will receive instructions anddata from a read-only memory and/or a random access memory. Generally, acomputer will include one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM disks. Any of the foregoing canbe supplemented by, or incorporated in, ASICs (application-specificintegrated circuits) and other forms of hardware.

The term “module” is not meant to be limited to a specific physicalform. Depending on the specific application, modules can be implementedas hardware, firmware, software, and/or combinations of these.Furthermore, different modules can share common components or even beimplemented by the same components. There may or may not be a clearboundary between different modules.

What is claimed is:
 1. A computer-implemented method for selectinglocations of focal planes for a multi-focal display in order for themulti-focal display to render actual content from a three-dimensionalscene, the method comprising: analyzing the actual content from thethree-dimensional scene to be rendered by the multi-focal display;selecting the locations of the focal planes based on the analysis of theactual content, wherein selecting the locations of the focal planescomprises selecting the locations based on K-means clustering of thelocations of points in the three-dimensional scene; configuring themulti-focal display according to the selected locations of the focalplanes; and rendering the actual content on the multi-focal display insaid configuration.
 2. The method of claim 1 wherein the selectedlocations are non-uniformly spaced over an operating range of themulti-focal display.
 3. The method of claim 1 wherein analyzing theactual content and selecting the locations of the focal planes occurs inreal-time.
 4. The method of claim 1 wherein the multi-focal displayrenders scenes using not more than six focal planes.
 5. The method ofclaim 1 wherein analyzing the actual content comprises analyzing theactual content from the three-dimensional scene in a spatial domain, andselecting the locations of the focal planes comprises selecting thelocations of the focal planes based on the analysis in the spatialdomain.
 6. The method of claim 1 wherein analyzing the actual contentcomprises analyzing the actual content from the three-dimensional scenein a spatial frequency domain, and selecting the locations of the focalplanes comprises selecting the locations of the focal planes based onthe analysis in the spatial frequency domain.
 7. The method of claim 1wherein selecting the locations of the focal planes is further based onrendering requirements of the multi-focal display.
 8. The method ofclaim 1 further comprising selecting a number M of focal planes based onthe analysis of the actual content.
 9. The method of claim 1 wherein themulti-focal display is a near-eye multi-focal display.
 10. A multi-focaldisplay for rendering actual content from a three-dimensional scene byusing a plurality of focal planes at different locations, the locationsof the focal planes determined based on analysis of the actual contentof the three-dimensional scene to be rendered, wherein selecting thelocations of the focal planes comprises selecting the locations based onK-means clustering of the locations of points in the three-dimensionalscene, the multi-focal display configurable according to the selectedlocations of the focal planes and rendering the actual content in saidconfiguration.
 11. The multi-focal display of claim 10 wherein theselected locations are non-uniformly spaced over an operating range ofthe multi-focal display.
 12. The multi-focal display of claim 10 whereinanalyzing the actual content and selecting the locations of the focalplanes occurs in real-time.
 13. The multi-focal display of claim 10wherein the multi-focal display renders scenes using not more than sixfocal planes.
 14. The multi-focal display of claim 10 wherein analyzingthe actual content comprises analyzing the actual content from thethree-dimensional scene in a spatial domain, and selecting the locationsof the focal planes comprises selecting the locations of the focalplanes based on the analysis in the spatial domain.
 15. The multi-focaldisplay of claim 10 wherein analyzing the actual content comprisesanalyzing the actual content from the three-dimensional scene in aspatial frequency domain, and selecting the locations of the focalplanes comprises selecting the locations of the focal planes based onthe analysis in the spatial frequency domain.
 16. The multi-focaldisplay of claim 10 wherein selecting the locations of the focal planesis further based on rendering requirements of the multi-focal display.17. The multi-focal display of claim 10 wherein the multi-focal displayis a near-eye multi-focal display.
 18. A computer program product foruse with a computer, the computer program product comprising anon-transitory computer readable medium having a computer program codeembodied therein for selecting locations of focal planes for amulti-focal display in order for the multi-focal display to renderactual content from a three-dimensional scene, the computer program codeperforming the steps of: analyzing the actual content from thethree-dimensional scene to be rendered by the multi-focal display;selecting the locations of the focal planes based on the analysis of theactual content, wherein selecting the locations of the focal planescomprises selecting the locations based on K-means clustering of thelocations of points in the three-dimensional scene; configuring themulti-focal display according to the selected locations of the focalplanes; and rendering the actual content on the multi-focal display insaid configuration.
 19. The computer program product of claim 18 whereinthe selected locations are non-uniformly spaced over an operating rangeof the multi-focal display.
 20. The computer program product of claim 18wherein analyzing the actual content and selecting the locations of thefocal planes occurs in real-time.